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English for Academic Purposes (EAP Writing) is a compulsory course for English 
literature and language students at Obirin University. The first semester focuses on 
expository writing, typical of the TOEFL' writing exam. The second semester focuses 
on writing about literature. To facilitate their writing all students are provided with a 
user account for Criterion, a web-based essay management system. Essays submitted to 
Criterion are evaluated by computer, and returned to the writer in approximately one 
minute. Submitted essays are kept on the Criterion server, where they can be accessed 
and evaluated by teachers, and further revised by students. This paper will present 
an overview of the Criterion essay management system in the context of an English 
for Academic Purposes (EAP) Writing course. We will discuss student and teacher 
productivity, providing feedback on student writing, using corpus linguistic techniques 
in conjunction with Criterion, and learning outcomes. 
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T his paper will present an overview of the Criterion web-based essay management system 1 
in the context of an English for Academic Purposes (EAP) Writing course. We will discuss 
student and teacher productivity, providing feedback on student writing, using corpus linguistic 
techniques in conjunction with Criterion, and learning outcomes. 

Overview of the EAP writing course 

English for Academic Purposes (EAP Writing) is a compulsory course for approximately 180 
second-year English literature and language students at Obirin University. The course was 
started in 2004 to develop students’ academic writing skills, particularly expository essay 
writing. In class, students practice generating ideas, outlining and mapping, composing drafts, 
and revising. The first semester focuses on expository writing, typical of the TOEFL® writing 
exam. The second semester focuses on writing about literature. 

Overview of Criterion 

To facilitate and manage their writing all EAP students are provided with a user account for 
Criterion, a web-based essay management system developed by ETS, the company which 
creates the TOEFL® and TOEIC® tests. Essays submitted to Criterion are evaluated and scored 
by computer, and returned to the writer in approximately one minute. Criterion provides 
diagnostic feedback on grammar, punctuation, style, and organization. In addition, essays 
are given a score of 1 to 6, based on the criteria used in the TOEFL® Test of Written English. 
Submitted essays are kept on the Criterion server, where they can be accessed and evaluated 
by teachers, and further revised by students. Teachers can attach comments or corrections to 
essays which appear the next time a student logs on. 

Productivity 

Criterion can greatly increase both student and teacher productivity. During the first semester, 
each student wrote five essays with an average of 9 revisions per essay. The ease with which 
teachers can access essays in progress and type feedback allowed instructors to assign a much 
larger number of essays than would normally be possible with a class size of 28 students. In 
addition, the automatic scoring was motivating to students. Their natural desire to improve 
their score encouraged them to revise much more often than we have seen in pencil and paper 
based writing courses. 

Using Criterion to give corrective feedback 

This section will discuss ways in which teachers can use Criterion to build on their approach to 
second language (L2) writing, specifically as regards the provision of corrective feedback. 

Corrective feedback in L2 writing pedagogy 

Perhaps the most widely known method of providing feedback is using written comments. 

Robb, Ross and Shortreed (1986) examined Japanese university students’ writing in four 
different conditions of written feedback (overt correction, coding of errors, highlighting of 
errors, and providing error counts) and found that writing practice over time resulted in gradual 
increases in accuracy and complexity regardless of the feedback condition. While written 
comments seem unlikely to accelerate grammatical development, research suggests that teacher 
commentary can lead to effective revisions on drafts and promote writers’ self-editing skills 
(Ferris, 1997, 2001). 

Another method, student-teacher writing conferences, may be more likely to lead to 
interlanguage development because conferences encourage students to negotiate meaning 
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with their instructor. Ferris and Hedgcock (2005) provide an overview of writing conferences, 
allaying criticism and offering suggestions relevant to the logistics, content and dynamics of 
student-teacher writing conferences. 

Feedback in the EAP writing course 

The following suggestions are intended to show how Criterion can be used in a manner 
consistent with the approaches to error correction outlined above. 

Write comments 

Criterion enables instructors to attach comments to a particular essay, which may direct 
attention to important feedback, provide encouragement, and address essay content. All 
comments are retained at the end of each essay and provide a useful record of student progress, 
as well as a check on whether advice is being understood and followed. 

Insert pop-up notes 

Pop-up notes allow instructors to link their written commentary to particular parts of an essay 
using hypertext. Notes can be created and then stored in a pop-up notes library where they can 
be conveniently accessed and reused. On the other hand some instructors feel the Criterion 
response time is too slow which overrides the convenience of the pop-up library. 

Supplement Criterion feedback 

Criterion errs on the side of accuracy by disregarding numerous errors. In fact, Burstein, 
Chodorow and Leacock (2003) reported on a trial in which Criterion’s writing analysis tools 
identified only 40% of subject- verb agreement errors (see also Otoshi, 2005). Consequently it is 
important not to rely solely on Criterion’s reporting of errors. 

Conference with students 

As Ferris and Hedgcock (2005) note, “the computer writing lab can be an ideal setting for 
one-to-one teacher- student interaction” (p. 206). Computer lab sessions provide a chance for 
teachers to follow up on comments and notes and offer assistance while students continue 
working on their drafts. 

Using corpus linguistic techniques with Criterion 

This section will describe how corpus linguistic techniques are being used in conjunction with 
Criterion in the EAP Writing course. 

Setting up a learner writing corpus 

Since essays written using Criterion are already in electronic form, it is a fairly straightforward 
matter to set up a learner writing corpus. Using the Batch Print Reports function, student essays 
can be copied and pasted into .txt files, which can then be analysed using concordance software 
such as Wordsmith Tools. Using this procedure we have set up a corpus of essays which runs 
to more than 300,000 words and which covers five different TOEFL 8 topics. The student writing 
was not edited prior to inclusion in the corpus. 

Using the learner writing corpus for error analysis 

As well as providing a useful source of model essays with which to address essay structure and 
rhetorical features, this corpus is another tool for giving feedback on sentence-level errors. 
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The use of learner writing corpora to identify recurring errors has long been advocated by 
researchers such as Johns (1986), Tribble and Jones (1990), and Biber, Conrad and Reppen 
(1998, pp. 197-200) who include a case study in which error patterns in essays written by non- 
native speakers are investigated using corpus-based techniques. 

The learner writing corpus can be used to raise learner awareness of errors that Criterion 
has already identified, such as sentence initial usage of the connectives and and so. Lin (1999) 
reports positive results when students compare the frequency of these errors in their own 
writing with examples found in native speaker corpora. In addition the corpus can be used 
to highlight errors that Criterion finds problematic. A common example of this kind of error 
is word-type confusion: students, for example, may confuse the words “convenient” and 
“convenience”, using the former as a noun and the latter as an adjective. Work to create a set of 
corpus-based materials that focus on these kinds of recurring errors is ongoing. 



Results and Discussion 

Overall the EAP students’ writing ability improved significantly during the course. In a timed 
writing activity at the beginning of the course 18 percent of students received a score of five 
or six from Criterion while 65 percent of students scored five or six in a similar timed writing 
activity at the end of the course, as shown in Table 1. 



Table 1. Score frequencies 
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Post test 
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Percent 


Count 
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1 


1% 
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0% 
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4% 
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3% 


3 


50 


41% 


13 


11% 


4 


43 


36% 


26 


21% 


5 


15 


12% 


53 


44% 


6 


7 


6% 


25 


21% 



Qualitative data in the form of student feedback and teacher impressions also support the 
view that Criterion played a role in promoting successful learning outcomes. Both teachers and 
students were positive about the increase in productivity, the role of feedback, and access to 
error analysis. Student comments written in response to a prompt asking them to evaluate the 
software and the course suggest that both Criterion feedback and guidance from instructors 
were important. A typical comment follows: 

[By using Criterion] I can find my mistakes easily and correct it quickly. In addition, 
comments from our teacher were also helpful for me. Criterion is automatic advice 
but teacher’s comments offered us different information from Criterion. So I think 
both are important for me. 

Finally, while we tend to see Criterion as motivating to our students, one caveat is that the 
students who participated in this course were all English majors who, one could argue, already 
have a high degree of motivation. Additional classroom research in other teaching contexts 
would be the best way to investigate whether online writing software such as Criterion can 
motivate student writers. 
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